{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "03e3af8d-a850-49bd-9b0b-707b29ee320e",
   "metadata": {},
   "source": [
    "# Multimodal Ollama\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/multi_modal/ollama_multi_modal.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "This notebook shows you how to use our Ollama multimodal integration.\n",
    "\n",
    "Supports complete, stream_complete, chat, stream_chat methods (async support coming soon).\n",
    "\n",
    "Use on its own or plug into broader [multi-modal use cases](https://docs.llamaindex.ai/en/stable/use_cases/multimodal.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f49494ce-2162-490d-8d42-e3a39ecc498f",
   "metadata": {},
   "source": [
    "#### Define Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c8d5f34-7131-4470-879f-480c60f55250",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.multi_modal_llms import OllamaMultiModal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cac7cff1-f4ce-4b1e-b5f1-8b62fc2e4505",
   "metadata": {},
   "outputs": [],
   "source": [
    "mm_model = OllamaMultiModal(model=\"llava\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d720d38-3611-464a-ae8c-491b2dd2bf04",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-02-03 11:41:04--  https://res.cloudinary.com/hello-tickets/image/upload/c_limit,f_auto,q_auto,w_1920/v1640835927/o3pfl41q7m5bj8jardk0.jpg\n",
      "Resolving res.cloudinary.com (res.cloudinary.com)... 2606:4700::6813:a641, 2606:4700::6813:a741, 104.19.166.65, ...\n",
      "Connecting to res.cloudinary.com (res.cloudinary.com)|2606:4700::6813:a641|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 181517 (177K) [image/jpeg]\n",
      "Saving to: ‘jerry_images/test.png’\n",
      "\n",
      "jerry_images/test.p 100%[===================>] 177.26K  --.-KB/s    in 0.01s   \n",
      "\n",
      "2024-02-03 11:41:04 (14.6 MB/s) - ‘jerry_images/test.png’ saved [181517/181517]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget \"https://res.cloudinary.com/hello-tickets/image/upload/c_limit,f_auto,q_auto,w_1920/v1640835927/o3pfl41q7m5bj8jardk0.jpg\" -O jerry_images/test.png"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0c667a2-e0fc-4929-adf9-dfbc0017359b",
   "metadata": {},
   "source": [
    "#### Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "427443b3-0c26-4e52-ab5d-82e63839428b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.multi_modal_llms.generic_utils import load_image_urls\n",
    "\n",
    "image_urls = [\n",
    "    # \"https://www.visualcapitalist.com/wp-content/uploads/2023/10/US_Mortgage_Rate_Surge-Sept-11-1.jpg\",\n",
    "    # \"https://www.sportsnet.ca/wp-content/uploads/2023/11/CP1688996471-1040x572.jpg\",\n",
    "    \"https://res.cloudinary.com/hello-tickets/image/upload/c_limit,f_auto,q_auto,w_1920/v1640835927/o3pfl41q7m5bj8jardk0.jpg\",\n",
    "    # \"https://www.cleverfiles.com/howto/wp-content/uploads/2018/03/minion.jpg\",\n",
    "]\n",
    "\n",
    "image_documents = load_image_urls(image_urls)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f47de9bf-1a31-4b12-b388-a23fc5905c22",
   "metadata": {},
   "source": [
    "#### Completion (Non-Streaming/Streaming)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96ebffbe-be55-48fc-9cc9-305654c9e20a",
   "metadata": {},
   "outputs": [],
   "source": [
    "complete_response = mm_model.complete(\n",
    "    prompt=\"Tell me more about this image\", image_documents=image_documents\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "509de1e2-319d-4f30-81ce-212a9a9025e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " The image shows the ancient Greek landmark known as the Colosseum, located in Rome, Italy. It is lit up with colorful lights and appears to be illuminated against a night sky. The Colosseum is a distinctive oval structure that has been used for various purposes over the centuries, including gladiatorial contests and public events. The colors of light suggest they could be representing national colors, such as those of Italy (red), white, and green. The photo captures the grandeur and historical significance of this iconic monument. \n"
     ]
    }
   ],
   "source": [
    "print(str(complete_response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "483b3222-11ee-4ab1-9e3c-f223e55aa7c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " This is an image of the Colosseum, a famous landmark located in Rome, Italy. It's illuminated at night with colorful lights, which gives it a festive appearance. The Colosseum is a significant historical and architectural structure that was used for gladiatorial contests and other public spectacles during the Roman Empire. "
     ]
    }
   ],
   "source": [
    "response_gen = mm_model.stream_complete(\n",
    "    prompt=\"Tell me more about this image\",\n",
    "    image_documents=image_documents,\n",
    ")\n",
    "for r in response_gen:\n",
    "    print(r.delta, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a9f9c44-659c-4e71-9728-1388e9160c09",
   "metadata": {},
   "source": [
    "#### Chat (Non-Streaming/Streaming)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af3203ce-4a43-44e2-b70b-17df390d9dc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# chat\n",
    "from llama_index.llms import ChatMessage, MessageRole\n",
    "\n",
    "image_bytes_io = [d.resolve_image() for d in image_documents]\n",
    "\n",
    "chat_response = mm_model.chat(\n",
    "    [\n",
    "        ChatMessage(\n",
    "            role=MessageRole.USER,\n",
    "            content=\"Tell me more about this image\",\n",
    "            additional_kwargs={\"images\": image_bytes_io},\n",
    "        )\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1839bc6-3b02-4ff3-a0c4-dad3e7f995a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "assistant:  This is an image of the Colosseum, also known as the Flavian Amphitheatre, which is a renowned landmark located in Rome, Italy. The Colosseum is one of the most famous and well-preserved ancient structures in the world. It was built during the Roman Empire and was used for public entertainment such as gladiatorial contests, reenactments of battles, and dramas based on classical mythology.\n",
      "\n",
      "The structure has a distinctive elliptical shape with tiered seating for spectators, which is clearly visible in this image. The photo is taken at night, and the Colosseum is illuminated by colorful lights that highlight its arches and the overall outline of the building. The colors of the lights correspond to those of the Italian flag, symbolizing a sense of national pride or celebration.\n",
      "\n",
      "The architecture and design of the Colosseum are indicative of the engineering prowess of the Romans during their peak period. It has become an iconic symbol of Roman civilization and continues to be a popular tourist attraction in Rome. \n"
     ]
    }
   ],
   "source": [
    "print(str(chat_response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "977f1d6a-bde2-41be-8229-684f21f44014",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " The image shows the Colosseum, a well-known landmark and amphitheater in Rome, Italy. It is illuminated with colorful lights, which appear to be red, green, and yellow, possibly indicating a special event or celebration, such as a national holiday given the colors of the Italian flag (red, white, and green). The Colosseum is captured at night under a clear sky, which adds to its dramatic presentation. "
     ]
    }
   ],
   "source": [
    "# stream chat\n",
    "from llama_index.llms import ChatMessage, MessageRole\n",
    "\n",
    "\n",
    "image_bytes_io = [d.resolve_image() for d in image_documents]\n",
    "\n",
    "chat_gen = mm_model.stream_chat(\n",
    "    [\n",
    "        ChatMessage(\n",
    "            role=MessageRole.USER,\n",
    "            content=\"Tell me more about this image\",\n",
    "            additional_kwargs={\"images\": image_bytes_io},\n",
    "        )\n",
    "    ]\n",
    ")\n",
    "for r in chat_gen:\n",
    "    print(r.delta, end=\"\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llama_index_v2",
   "language": "python",
   "name": "llama_index_v2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
